WebToData

The WebToData program is designed to easily remove HTML code from HTML files. It can also process HTML files to find and extract data and create database ready files. Files processed by WebToData can be edited in the program or saved and edited using a word processor or database program.

Version 2 of WebToData includes powerful utilities to convert web tables to character separated text that can then be transferred to tables in word processors, spreadsheets or databases.

HTML lists are converted to a list closely matching that of the list type specified in the HTML code.  For example numbered lists are translated to number items in text format. Check WebToData Frequently Asked Questions for tips and information on how to get more out of this program.

The program provides 4 output options. The user can select which option is the most useful for their purposes.

The four output options are:

1/	Original HTML  window

2/	Formatted HTML window

3/	Text File window

4/	Formatted Text window

The process used to remove HTML code form files is:

1.	Open a valid HTML file

1.	Process the HTML Code

1.	Remove extra lines

1.	Save the desired output


Open a Document

WebToData lets you open HTML, Rich Text and text files. The program is designed to remove HTML code from HTML files so the majority of files opened will be in the HTML format.

When the program opens a file it is inserted into the Original HTML file window regardless of the file’s format.  This window is the active window for all newly opened files.

Opening a file

1.	Choose Open from the File menu. Alternatively use the short cut keys Ctrl-O or the open file icon [image].

1.	Select the required file

1.	Press the Open button.


Save a Document

WebToData lets you save documents in HTML, Rich Text or Text formats.  When saving documents the program will save the contents of the active window.  This means if the Formatted HTML window is active, the contents of this window will be saved to file.

Care should be taken when saving files that the required format is selected before saving the file.  A file saved in Rich Text format will not read as desired in a Web Browser.

Saving a file

1.	Select Save from the File menu. Alternatively use the short cut keys Ctrl-S or the save file icon  [image].

1.	Type a file name

1.	Press the Save button with the mouse curser.


The Active window is the window showing current information.


New Document

While it was not intended to be used as an instrument used to create HTML documents WebToData may be used to create and edit HTML, Rich Text and text files.

To create a new document

1.	Select New from the File menu.  Alternatively press Ctrl-N or the New icon.


Exit
Exit WebToData

To exit WebToData simply press the Ctrl-Q keys or select Exit for the Files menu.


Remove HTML Code

To process the current HTML document press the Ctrl-H keys or alternatively press the Remove HTML icon on the toolbar.

Once the Remove HTML code button has been pressed the program will automatically remove all HTML code form the original HTML file.  At no time does the program change the original file during processing.  The Program will then display the results in the text file window.

WebToData converts tables to character separated text that can then be transferred to tables in word processors, spreadsheets or databases.  HTML lists are converted to a list closely matching that of the list type specified in the HTML code.  For example numbered lists are translated to number items in text format.

Once the HTML file has been processed you can save the text file, formatted HTML file or the original HTML file.


 WebToData has the ability to insert characters that separate cells found in HTML tables.  Tables found within files processed by WebToData can then converted to database tables, spreadsheets or tables in word processors.


Text File Window

The text file window displays the results of removing the HTML code from the HTML file. In it you will have the opportunity to edit the text file and compare it with the original HTML file.

If you are transferring a web table to a database use this window to view all data found in the web page tables.

You can elect to save the file or remove blank lines from the file.


Original HTML Window

The original HTML window displays the HTML file in its original state.  It is possible to edit files in this window if desired.   Files can be changed and saved in this window.

 The main purpose of this window is to provide the opportunity to compare any processed files with the original version.


Formatted HTML Window

In order to verify the formatting of HTML files WebToData checks the original HTML file. As it checks the layout of code in a file it formats the file into an easier to read style.  The result is a file that is usually much easier to decipher the logic than the original. This file can used to compare and check the contents of the processed text file with the contents of the HTML file.

Formatted HTML files can be saved and then viewed in a Web browser


Formatted Text Window

WebToData has the ability to remove extra or unwanted blank lines from the text file.

The Formatted Text window shows the file in its final format.  To remove extra lines use the Remove Extra Lines button.

Once the extra blank lines have been removed the program automatically displays the Formatted Text window.


Remove Extra Blank Lines
Remove Blank Lines

WebToData has the functionality to remove extra blank lines encountered in normal HTML coding. The decision of whether or not to remove the lines is entirely up to you.  The Text window displays the HTML file with code removed but with line breaks as found in the HTML code.  If your file has extra blank lines that you wish to remove use the Remove Extra Lines button.


Tool bar

The Tool bar enables you to access most WebToData functions using the mouse.


Include WWW Links

WebToData provides you with option to include links to other pages as part of the text output.  The Include WWW Links box must be ticked before WebToData includes links to other pages in its final output.

To include WWW links:

1.	Click the Include WWW Links box so it shows a tick. If the box already shows a tick it is already including  WWW links

Alternatively

1.	Open the Option Form by selecting the Options from the Options menu


Include Mail Links

WebToData provides you with option to include links to other mail information as part of the text output.  The Include Mail Links box must be ticked before WebToData includes links to other mail in its final output.

To include Mail links:

1.	Click the Include Mail Links box so it shows a tick. If the box already shows a tick it is already including  Mail links

Alternatively

1.	Open the Option Form by selecting the Options from the Options menu


Include Other Links

WebToData provides you with option to include links to other information as part of the text output.  The Include Other Links box must be ticked before WebToData includes links to other information in its final output.

To include other links:

1.	Click the Include Other Links box so it shows a tick. If the box already shows a tick it is already including  Other links

Alternatively

1.	Open the Option Form by selecting the Options from the Options menu


Length of Text Line

WebToData gives you the flexibility to select a maximum length for the lines of text converted from HTML to text format.  For example if you wish the maximum length to be 85 characters this number should be input into the appropriate section of the Options Form.

If you make changes it is possible to revert back to the Default value of 80 characters.

IMPORTANT NOTE

When using any of the data extraction tools it is recommended that the length of the text line be set to a value greater than 80.  A setting of 200 is recommended.  For large data files this can even be extended to 2000 or more.


Table Cell Separator

It is possible to specify the character used to separate the table cells found in HTML formatted tables.  This is one of the most powerful features of HTM Stripper the ability to determine the contents of tables and separate the contents.  By using this process it is possible to convert these tables to database tables.

A single area where a row and column meet.


Options: Command OK

Use this button to apply the settings and close the form.


Options: Command Apply

Use this button to apply the information in the Options form without closing the form


Transfer Web Table to Data

To transfer web data into a database ready file complete the following actions.  If you need further information on setting up Data Transfer Options select the Tutorial from the help menu.

1.	Open a valid HTML file

1.	Select the most appropriate Data Matching option

1.	Select ‘Transfer Web Page to Data from the Actions menu.

Alternatively

Press Ctrl+T or use the toolbar

1.	It is important to check the output.  If the data does not match what you expect you may need to use another data Matching Option.

Check Data Extraction Options for more details of the tutorial


Data Finding Options
Data Extraction Options

WebToData includes a number of powerful choices for finding and extracting data found in web pages.

Before using any of the data extraction tools please ensure that the text length is set to a value large enough to accommodate the size of the record that WebToData will find.

To cater for the number of different formats that can be used to display data in web pages WebToData provides a number of different options.

The different options are:

•	Data Sensing.

This is the default setting that uses artificial intelligence to find data in web pages.  When extracting data from web pages it is recommended that this setting be used. If it is not successful, one of the other options can be selected.

WebToData provides an easy way to determine if the data extracted from the web page is valid. To decide if the data is valid switch between the Text File view and the Formatted Text view.  The database file is stored in the Formatted Text view.

IMPORTANT INFORMATION

Even if Data Sensing is not successful it is recommended that this tool be used first to determine the structure of the web page data. WebToData is designed to provide a number of different views of the web page being processed. If the final results are not as expected, a different view of the data can be used to determine the custom settings for User Defined Data or Pattern Matching configurations. should be used t

•	User Defined Data

This option should be used if both of the following conditions are met.

1)	Data Sensing does not provide the desired results.

2)	When viewing the results in the Text File view it is determined that each database record is confined to one line.

For more information move to User Defined Data

•	Pattern Matching

This option should be used if the following conditions are met.

1)	Data for a single record is stored on multiple lines.  This means that both Data Sensing and User Defined Data will not give you the desired results.

2)	When viewing the results of a trial run in the Text File view there is a definite pattern in the way that the data is stored.


Data Sensing

WebToData’s Data Sensing mode is used to extract data from web pages.  It is a powerful technology that uses artificial intelligence to find data in web pages.  It is recommended that this mode be used first.

Once a file has been processed using Data Sensing it is important to check the Data Validity

If you are not getting the expected results you can customize the information used by WebToData to find and extract data.

Note

Before using any of the data extraction tools please ensure that the text length is set to a value large enough to accommodate the size of the record that WebToData will find.


Data Validity

When you Transfer Web Page to Data you will need to use two important views.

1.	The first important view is the Text File view.  All table data found in the processed web page is displayed in this view.

1.	The second view is the Formatted Text view.  In this view you will find the data ready to be saved for use in a database or spreadsheet.

It is important to recognise that these two views will be used by you to make decisions about the validity of the data.  To access the validity of data, switch between the two views to determine if the data found in the Formatted Text is displaying the expected results.

Further information on how to set up WebToData to find data in web pages is available from the Tutorial under the Help menu.

Note

Before using any of the data extraction tools please ensure that the text length is set to a value large enough to accommodate the size of the record that WebToData will find.


Data Pattern Matching

There will be times when the structure of data found in web pages makes it difficult to find and extract. An example of data difficult to find is when data is spread out over a number of lines in a web page. WebToData provides a Pattern Matching tool as the means for solving this common problem.

Before Setting Up Pattern Matching.

1.	Open a valid HTML File containing data.

1.	Make sure the ‘Data Sensing Option’ is set to on. To do this just make sure the Use Data Sensing option from the Data menu has a tick.

This is done to run a test to check if Data Matching is required.

1.	Run the ‘Transfer Web Page to Data’ command.

1.	Check the Data Validity

1.	If the data is not presented correctly, check the data pattern in the Text File window.

A fair amount of user interpretation is required to set up data matching correctly. It may take a number of trials to work out the exact pattern for the data been transferred. This is one area were practice will improve initial success rates.

The Tutorial is designed to make setting up pattern matching easy and accurate.

Setting up Pattern Matching

1.	Select ‘Pattern Matching Options’ from the ‘Options’ menu

1.	Enter the number of lines to process.

1.	Enter the number of delimiters.

Note

Before using any of the data extraction tools please ensure that the text length is set to a value large enough to accommodate the size of the record that WebToData will find.


Number of Lines to Process

When using the Pattern Matching option you will have to identify how many lines are in a pattern. Information on running a test before setting up Pattern Matching is in the Data Matching Section.

After running a test it is straightforward to work out the pattern for the data found in the web page. The first thing to configure is how many lines are in the pattern.  If there four lines in a pattern (make sure you count blank lines) type 4 into the ‘# Lines to Process’ box.

Further information on configuring WebToData for Data Extraction is available in the tutorial.


User Defined Data Settings

User Defined Data Settings should be used when the following conditions are met.

1/	The data for a record is found on one line in the HTML table.  If the data is stored over multiple lines, Data Pattern Matching should be used.

2/	The Data Sensing does not give the desired results.

Further information on User Defined Data Setting is available in the tutorial.

Note

Before using any of the data extraction tools please ensure that the text length is set to a value large enough to accommodate the size of the record that WebToData will find.


Configuring User Defined Data Settings

User Defined Data Options can be configured to find data that meets certain conditions. To setup WebToData to find User Defined Data complete the following.

1/	Set User Defined Data Delimiters to on.

2/	Enter the number of delimiters that WebToData needs to look for when processing data.

Further information is available in the tutorial.


Option Form
Options Form

The Options Form provides a means of configuring various options for WebToData.  The options that can be configured  include:

1/	Text Options

2/	Hyper Links

3/	Clean Up

4/	User Defined Data Settings

5/	Pattern Matching

6/	Extended Pattern Matching

The various options are available by selecting the appropriate tab in the Options Form.


Hyper Link Options
Hyper Links Options

WebToData can be configured to manage the following Hyper Links options.

1/	Include Mail Links

2/	Include WWW Links

3/	Include Other Links


Clean Up Options
Text Options

WebToData can be configured to set up the table delimiter, the length of the line in characters and the character used to indicate lists.

Note

If you are using the program for extracting data from html files, the program automatically defaults to using a comma to separate data.


Text Options
Clean Up Options

To remove extra text or code, insert the text or code that you want WebToData to find.

WebToData can be configured to manage the following text options.

1/	Text Length

2/	Table Cell Separator


Clean Up

WebToData can look for and clean up leftover code found in html documents. To clean up documents go to the Clean Up section of the Options Form, insert the text or code that you want WebToData to find.

The program will automatically remove the code inserted in the Options Form.  To stop WebToData from removing extra code just remove it from the Clean Up section of the Options Form.


User Defined Data Delimiters On

When ticked, WebToData is set to process using User Defined Data Settings.


Remember User Defined Data Delimiters Settings
Remember User Defined Data Settings

When ticked the program will remember the custom User Defined Data Settings.

Warning if this boxed is not ticked, the program will store the settings from the last time this setting was activated.


Number of Data Delimiters - Pattern Matching
Number of Data Delimiters – Pattern Matching

When managing Pattern Matching you must insert the number of delimiters the program will search for in each line when looking for valid data in html documents.

For more information on setting up WebToData for Pattern Matching data extraction use the tutorial.


Number of User Defined Data Delimiters

When managing User Defined Data Settings you must insert the number of delimiters the program will search for when looking for valid data in html documents.

For more information on setting up WebToData for data extraction use the tutorial.


Pattern Matching On

When managing Pattern Matching you must tell WebToData to process documents using Pattern Matching.

This is done by clicking the Pattern Matching On box so it has a tick.


Blank Line

This box is ticked if the line number that this box refers to in the pattern is blank.


Text in Line - Pattern Matching
Text in Line – Pattern Matching

This box is ticked if the line number that this box refers to in the pattern has text.


Add Line - Pattern Matching
Add Line – Pattern Matching

This box is ticked if the line number that this box refers to in the pattern has text that must be added to the finished database.


Remove both HTML Code and Extra lines

WebToData has the ability to remove the HTML code from HTML documents and remove extra lines in a single process.

To run both functions consecutively do the following:

1/	Choose Remove both Code and Lines from the Options menu

alternatively

1/	Press the Ctrl and R keys together.

For fast removal of html code and lines from a single document this is the quickest method.

To remove only html code, use the Remove HTML Code function.


Clear All - Pattern Matching
Clear All – Pattern Matching

Click this button to clear all values for Pattern Matching.


Printing

WebToData can print the output of any window.  It automatically prints the active window.

To print select Print from the File menu.


Pattern Matching Options

Pattern Matching requires a number of options to be set up.  To learn more about setting up Pattern Matching select one of the following areas.

Turning Pattern Matching On

Number of Delimiters

Indicating Blank Lines

Indicating that text is in the line

Further information is available from the WebToData tutorial


Multi File Processing

WebToData can be set up to automatically process many files in a single session. There are two ways that WebToData can process files.

1/	Create an individual file for each file processed. For example if 5 files are set up for processing, WebToData will automatically create 5 corresponding files.

2/	Merge multiple files into a single file. The ability to merge many files into a single file is a particularly useful feature if you are creating a database based on many HTML files.

Selecting Files to Process

1/	Select the input folder.  This is the folder containing the HTML files you are going to process.

2/	Select the files you are going to process. To select files for processing just use the CONTROL or SHIFT key in conjunction with mouse clicks.  Using the SHIFT key will enable you to select many files in one click.  Using the CONTROL key will enable to pick out individual file’s from anywhere in the list.

Select an Action

There are 3 options to select from when Multi File Processing.

1/	Remove HTML Code – This action processes the web page returning it to text file format without HTML Code.

2/	Remove HTML Code and Lines – In addition to the processing carried out by removing HTML Code, this action removes extra lines.

3/	Transfer Web to Data – This action searches for data contained within HTML pages and transfers the data to a new file. Before using this option you must set the type of data processing to be carried out using the Data menu.  This option will use the action set in the Data menu.

IMPORTANT NOTES

Before proceeding with multi file processing using the “Transfer Web to Data” format it is advised that one file from the list be processed to ascertain and confirm the required settings for the data transfer.

When transferring multiple data files into a single file it is important that the source files be based on the same data structure. WebToData will merge files with different data structures, but the end result will not be satisfactory for database implementations.

Select a File Option

There are 2 options to select from.

1/	Multiple Files – This option sets WebToData to create a separate file for each selected file.

2/	Single File - Using this option will result in WebToData merging the results of many files into a single file.

Select an Output Folder

You have the option to select an output folder.  To select a folder either “double click” on the desired folder or using the mouse, press the “Select Folder” button.  If you do not select an output folder, the source folder will be used.

Enter an Output File Name

When the Single File Option is selected an “Output File” input box becomes visible.  The program will prompt you for a valid file name before proceeding.


Extended Pattern Matching

Extended Pattern Matching extends the number of lines that can be processed when extracting data from the html files.  Standard Pattern Matching has a limit of 5 lines. Extended Pattern Matching has a limit of 200 lines.

Extended Pattern Matching is set-up using the same principles as Standard Pattern Matching.  So it is important to acquaint yourself with the principles behind Standard Pattern Matching before attempting to set-up Extended Pattern Matching.

Extend Pattern Matching is the most complex and difficult data extraction tool to set-up and use.  It will find any data that the other tools will find.  It should, because of its complexity, be used as a last resort.

The only time that Extended Pattern Matching should be used is if the number of lines that need to processed to make up a data file extend beyond the limit of standard Pattern Matching. Of course standard Pattern Matching should only be used if the data is beyond the reach of the other simpler to use tools.  The settings for Extended Pattern Matching are found in Options.

Setting up Extended Pattern Matching

Setting up Extended Pattern Matching is similar to setting up command line settings.

Number of Lines

Number of Delimiters in Line

Add Text To Data File

Text In Line

Blank Line

Sample of Setting Up Extended Pattern Matching

The numbers above each of the settings boxes provide useful information designed to make it easier to set up Extended Pattern Matching.  The number on the right indicates the total number of the lines that are to be processed.  The number on the left indicates the current position of the cursor within the set-up box.


Extended Pattern Matching - Number of Delimiters in Line
Extended Pattern Matching – Number of Delimiters in Line

Setting up the Number of Delimiters for Extended Pattern Matching is similar to setting up the number of delimiters for Standard Pattern Matching.  Inserting numbers that indicate how many delimiters are in each line sets this up.

See an example


Extended Pattern Matching - Add Text to Data File
Extended Pattern Matching – Add Text to Data File

WebToData can set up to add specific lines of data to the data file. In others words there is no need to add a line simply because it is in the data set. Setting up whether or not “text is added to a data file” requires a fair amount of user analysis of the data file. You have the ultimate control over whether or not a line is added to a file.  The example provides a sample data set and how this feature can be set up.

See an example


Extended Pattern Matching - Text In Line
Extended Pattern Matching – Text In Line

When setting up WebToData for Extended Pattern Matching the program needs to know if there is text in the line that will be processed as part of a data set. You insert either a one (1) to indicate that there is text in the line or a zero (0) to indicate that there is no text in the line.  Check the example to find out further details.

See an example


Extended Pattern Matching - Blank Line
Extended Pattern Matching – Blank Line

When setting up WebToData for Extended Pattern Matching the program needs to know if there is a blank line that will be processed as part of a data set. You insert either a one (1) to indicate that there is a blank line or a zero (0) to indicate that there is text in the line.  Check the example to find out further details.

See an example


Extended Pattern Matching - Sample Set Up
Extended Pattern Matching – Sample Set Up

The following sample data will be used as an example for setting up Extended Pattern Matching.

NOTE:  The delimiter used for this example is an asterisk.

Stock*

Stock Id: 1234* Number in Stock: 789* Restock Level: 45*

Stock Description:  Handle*

Stock Location: London* Stock Location ID: 098*,

Stock Suppliers: Bourke’s, Tom’s, Henderson’s, Fred’s*

Alternative Stock Id: 1235*

Number of Lines to Process

In the example above there are 11 lines of data including blank lines.  Note always include any blank lines that are included in the data set.

The figure of 11 would be inserted in the Number of Lines to Process box. Alternatively we might decide that we do not need to include the line containing “Stock*” or the following blank line. We could then tell WebToData to process 9 lines.  For our example we will stick with 11 lines to process.

Number of Delimiters in Line

The Number of Delimiters in Line is more complex to set-up. Looking at the first line we can see that there is one delimiter in the line.  We start by adding the number one to the Number of Delimiters in Line box.  The next line contains no delimiters.  Therefore we put zero(0) next to the number one(1) that we just inserted.  All the numbers going into all boxes are separated by a space.  There is no need to use a comma or any other delimiter to separate the numbers going into these boxes.  In fact to do so will introduce an error.

The numbers going into the Number of Delimiters in Line box will look like this:

1 0 3 0 1 0 2 0 1 0 1

It is important to note that the number of numbers should correspond with the number of line to process.  If we count the example above you will note that there are 11 numbers.  It is also important to note that the number of delimiters placed should correspond exactly with the number of delimiters found in the data file.  Any difference will cause the program not to process the required files.

Add Text to Data File

We can control what data is added to the data file. We control what data is added by inserting either a zero(0) or a one(1).  Essentially the zero(0) stands for false – do not add the line to the file. Or one(1) - add the line to the data file. Looking at the sample data it is unlikely that the first line will be needed in any data file.  Therefore a zero(0) would be added as the first figure.  The second line is blank, we don’t want it so we add a zero again. The next line contains text and we want it, so we add a one(1).  The complete set up for the “Add Text to Data File” would look like this:

0 0 1 0 1 0 1 0 1 0 1

Note we have 11 numbers that correspond exactly with the number of lines to process.

Text in Line

The program also needs to know if there is text in the line to process.  Again true of false is indicated by a one(1) or a zero(0).  The first line contains text so the first number in this set up box would be one(1).  The second line is blank so we insert a zero (0).  The complete set up for this would look like the following:

1 0 1 0 1 0 1 0 1 0 1

Blank Line

The program needs to know if lines are blank or not. Again this indicated by a one(1) for true of zero(0) for false.  The set up of this box follows the same format as the previous boxes.

0 1 0 1 0 1 0 1 0 1 0

It is important to ensure that the number of numbers correspond with the number of line to process.


WebToData Tips

WebToData FAQs

Q1) 	When using the Multi-File Processing function I find that some files are not added to the data file.  I have checked the file structure and to the best of my knowledge all html files have the same structure.

A1)	Some html files may have data already separated by commas.  For example one file may have an address be set out as street, state, zip.  Another html file may have the same data set out as street state or zip.  If the comma delimiter is used the extra commas found in the file will cause a problem.  To overcome the problem, use another delimiter.  The Tab delimiter is the most reliable because all tabs are removed from the file during preliminary processing.

Q2)	I want to use the Tab delimiter to separate data in my data file. However, because of the file structure I need to use Extended Pattern Matching.  How do I work out the data structure for the Options box when I cannot see the delimiters?

A2)	Change the delimiter to a type that you can see.  You can change your delimiter type at any time.

Q3)	Throughout the Help file I see reference to setting up Pattern Matching and Extended Pattern Matching.  How to do I get WebToData to show me information I need to set up these features?

A3)	By opening a file and choosing to run the  “Transfer Web Page to Data” item from the Action menu you will get WebToData to attempt to process the file.  If the “Formatted Text” view is blank then the action was unsuccessful.  However, by selecting the “Text File” you see the data structure for the file.  Use this view to set up data matching options.

Q4)	Sometimes the program appears to truncate data. Is there a setting that can be changed to avoid this?

A4)	When processing data files it does not hurt to extend the “Text Length” to a large number.  I have when necessary extended the length to 2000.

Q5)	At times I wish to remove special characters or words found in the text.  How do I do this?

A5)	Use the “Clean Up” options to enter any valid character or group of characters that can be removed from the text stream.

Q6)	Can the multi-file function create a separate file for each file processed?

A6)	Multi-file processing can be set to either merge all processed files into one large file or it can create an individual file for each file processed.